
import Head from 'next/head'

<Head>
  <script>
    {
      `(function() {
         var _hmt = _hmt || [];
(function() {
  var hm = document.createElement("script");
  hm.src = "https://hm.baidu.com/hm.js?e60fb290e204e04c5cb6f79b0ac1e697";
  var s = document.getElementsByTagName("script")[0]; 
  s.parentNode.insertBefore(hm, s);
})();
       })();`
    }
  </script>
</Head>

![LangChain](https://pica.zhimg.com/50/v2-56e8bbb52aa271012541c1fe1ceb11a2_r.gif)



Deep Lake
===========

> [Deep Lake](https://docs.activeloop.ai/) 是一个多模态的向量存储库，存储嵌入和它们的元数据，包括文本、json、图像、音频、视频等。它会在本地、您的云存储或Activeloop storage上保存数据。 它能执行包括嵌入和它们的属性的混合搜索。

本教程展示了与 `Deep Lake` 相关的基本功能。虽然 `Deep Lake` 可以存储嵌入，但它能够存储任何类型的数据。 它是一个具有版本控制、查询引擎和流式数据加载器的完整的无服务器数据湖，可供深度学习框架使用。

更多信息，请查看深度湖泊 [文档](https://docs.activeloop.ai) 或 [api 文档](https://docs.deeplake.ai)

```python
!pip install openai deeplake tiktoken

```

```python
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import DeepLake

```

```python
import os
import getpass

os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')
embeddings = OpenAIEmbeddings()

```

```python
from langchain.document_loaders import TextLoader

loader = TextLoader('../../../state_of_the_union.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()

```

在 `./deeplake/` 上本地创建数据集，然后运行相似性搜索。

Deeplake+LangChain的集成在底层使用Deep Lake数据集，因此`dataset`和`vector store`可以互换使用。 


要在自己的云中或Deep Lake存储中创建数据集，请[根据需要调整路径](https://docs.activeloop.ai/storage-and-credentials/storage-options)。

```python
db = DeepLake(dataset_path="./my_deeplake/", embedding_function=embeddings)
db.add_documents(docs)
# or shorter
# db = DeepLake.from_documents(docs, dataset_path="./my_deeplake/", embedding=embeddings, overwrite=True)
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)

```

```python
/home/leo/.local/lib/python3.10/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.3.2) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.
  warnings.warn(

```

```python
./my_deeplake/ loaded successfully.

```



```python
Evaluating ingest: 100%|██████████████████████████████████████| 1/1 [00:07<00:00

```

```python
Dataset(path='./my_deeplake/', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape      dtype  compression
  -------   -------   -------    -------  ------- 
 embedding  generic  (42, 1536)  float32   None   
    ids      text     (42, 1)      str     None   
 metadata    json     (42, 1)      str     None   
   text      text     (42, 1)      str     None   

```

```python
print(docs[0].page_content)

```

```python
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. 

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. 

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. 

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

```

Later, you can reload the dataset without recomputing embeddings

```python
db = DeepLake(dataset_path="./my_deeplake/", embedding_function=embeddings, read_only=True)
docs = db.similarity_search(query)

```

```python
./my_deeplake/ loaded successfully.

```



```python
Deep Lake Dataset in ./my_deeplake/ already exists, loading from the storage

```

```python
Dataset(path='./my_deeplake/', read_only=True, tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape      dtype  compression
  -------   -------   -------    -------  ------- 
 embedding  generic  (42, 1536)  float32   None   
    ids      text     (42, 1)      str     None   
 metadata    json     (42, 1)      str     None   
   text      text     (42, 1)      str     None   

```

Deep Lake目前是单写多读的。设置 `read_only=True`可帮助避免获取写锁。

检索问答[#](#检索问答 "Permalink to this headline")
-------------------------------------------------------------------------------------------

```python
from langchain.chains import RetrievalQA
from langchain.llms import OpenAIChat

qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model='gpt-3.5-turbo'), chain_type='stuff', retriever=db.as_retriever())

```

```python
/home/leo/.local/lib/python3.10/site-packages/langchain/llms/openai.py:624: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
  warnings.warn(

```

```python
query = 'What did the president say about Ketanji Brown Jackson'
qa.run(query)

```

```python
'总统提名Ketanji Brown Jackson担任美国最高法院法官。 他将她描述为一名前私人执业的顶级诉讼律师，一名前联邦公共辩护人，一个共识建设者，并来自公立学校教育者和警察的家庭。他还提到自她被提名以来，她得到了广泛的支持。'

```

基于元数据的属性筛选[#](#基于元数据的属性筛选 "Permalink to this headline")
-------------------------------------------------------------------------------------------------------------

```python
import random

for d in docs:
    d.metadata['year'] = random.randint(2012, 2014)

db = DeepLake.from_documents(docs, embeddings, dataset_path="./my_deeplake/", overwrite=True)

```

```python
./my_deeplake/ loaded successfully.

```

```python
Evaluating ingest: 100%|██████████| 1/1 [00:04<00:00

```

```python
Dataset(path='./my_deeplake/', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (4, 1536)  float32   None   
    ids      text     (4, 1)      str     None   
 metadata    json     (4, 1)      str     None   
   text      text     (4, 1)      str     None   

```



```python
db.similarity_search('What did the president say about Ketanji Brown Jackson', filter={'year': 2013})

```

```python
100%|██████████| 4/4 [00:00<00:00, 1080.24it/s]

```

```python
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.   Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.   One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.   And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
 Document(page_content='And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.   As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.   While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.   And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.   So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.    First, beat the opioid epidemic.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013})]

```

Choosing distance function[#](#choosing-distance-function "Permalink to this headline")
---------------------------------------------------------------------------------------

欧几里得距离的距离函数`L2`，核距离的距离函数`L1`，最大的l-infinity距离`Max`，余弦相似度`cos`，点积`dot`

```python
db.similarity_search('What did the president say about Ketanji Brown Jackson?', distance_metric='cos')

```

```python
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.   Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.   One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.   And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
 Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.   And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system.   We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.    We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.    We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster.   We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
 Document(page_content='And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.   As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.   While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.   And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.   So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.    First, beat the opioid epidemic.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
 Document(page_content='Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers.   And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.    That ends on my watch.   Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect.   We’ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees.   Let’s pass the Paycheck Fairness Act and paid leave.    Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty.   Let’s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jill—our First Lady who teaches full-time—calls America’s best-kept secret: community colleges.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012})]

```

最大边际相关性[#](#maximal-marginal-relevance "此标题的永久链接")
--------------------------------------------------

使用最大边际相关性

```python
db.max_marginal_relevance_search('What did the president say about Ketanji Brown Jackson?')

```

```python
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.   Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.   One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.   And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
 Document(page_content='Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers.   And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.    That ends on my watch.   Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect.   We’ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees.   Let’s pass the Paycheck Fairness Act and paid leave.    Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty.   Let’s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jill—our First Lady who teaches full-time—calls America’s best-kept secret: community colleges.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
 Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.   And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system.   We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.    We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.    We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster.   We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
 Document(page_content='And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.   As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.   While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.   And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.   So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.    First, beat the opioid epidemic.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013})]

```

删除数据集[#](#delete-dataset "此标题的永久链接")
------------------------------------

```python
db.delete_dataset()

```

如果删除失败，您还可以强制删除

```python
DeepLake.force_delete_by_path("./my_deeplake")

```



云上的Deep Lake数据集（Activeloop、AWS、GCS等)或在内存中[#](#deep-lake-datasets-on-cloud-activeloop-aws-gcs-etc-or-in-memory "此标题的永久链接")
-------------------------------------------------------------------------------------------------------------------------

默认情况下，Deep Lake数据集存储在本地。如果您想将它们存储在内存中、Deep Lake管理的数据库中或任何对象存储中，可以提供[相应数据集的路径](https://docs.activeloop.ai/storage-and-credentials/storage-options)。您可以从[app.activeloop.ai](https://app.activeloop.ai/)获取您的用户令牌

```python
os.environ['ACTIVELOOP_TOKEN'] = getpass.getpass('Activeloop Token:')

```

```python
# Embed and store the texts
username = "<username>" # your username on app.activeloop.ai 
dataset_path = f"hub://{username}/langchain_test" # could be also ./local/path (much faster locally), s3://bucket/path/to/dataset, gcs://path/to/dataset, etc.

embedding = OpenAIEmbeddings()
db = DeepLake(dataset_path=dataset_path, embedding_function=embeddings, overwrite=True)
db.add_documents(docs)

```

```python
Your Deep Lake dataset has been successfully created!
The dataset is private so make sure you are logged in!
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/davitbun/langchain_test
hub://davitbun/langchain_test loaded successfully.

```

```python
Evaluating ingest: 100%|██████████| 1/1 [00:14<00:00

```

```python
Dataset(path='hub://davitbun/langchain_test', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (4, 1536)  float32   None   
    ids      text     (4, 1)      str     None   
 metadata    json     (4, 1)      str     None   
   text      text     (4, 1)      str     None   

```

```python
['d6d6ccb4-e187-11ed-b66d-41c5f7b85421',
 'd6d6ccb5-e187-11ed-b66d-41c5f7b85421',
 'd6d6ccb6-e187-11ed-b66d-41c5f7b85421',
 'd6d6ccb7-e187-11ed-b66d-41c5f7b85421']

```

```python
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
print(docs[0].page_content)

```

```python
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. 

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. 

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. 

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

```

创建AWS S3上的数据集[#](#creating-dataset-on-aws-s3 "Permalink to this headline")
---------------------------------------------------------------------------------------

```python
dataset_path = f"s3://BUCKET/langchain_test" # could be also ./local/path (much faster locally), hub://bucket/path/to/dataset, gcs://path/to/dataset, etc.

embedding = OpenAIEmbeddings()
db = DeepLake.from_documents(docs, dataset_path=dataset_path, embedding=embeddings, overwrite=True, creds = {
   'aws_access_key_id': os.environ['AWS_ACCESS_KEY_ID'], 
   'aws_secret_access_key':  os.environ['AWS_SECRET_ACCESS_KEY'], 
   'aws_session_token': os.environ['AWS_SESSION_TOKEN'], # Optional
})

```

```python
s3://hub-2.0-datasets-n/langchain_test loaded successfully.

```

```python
Evaluating ingest: 100%|██████████| 1/1 [00:10<00:00
\

```

```python
Dataset(path='s3://hub-2.0-datasets-n/langchain_test', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (4, 1536)  float32   None   
    ids      text     (4, 1)      str     None   
 metadata    json     (4, 1)      str     None   
   text      text     (4, 1)      str     None   

```

```python

```

Deep Lake API [#](#deep-lake-api "Permalink to this headline")
-------------------------------------------------------------

您可以在db.ds上访问Deep Lake数据集。

```python
# get structure of the dataset
db.ds.summary()

```

```python
Dataset(path='hub://davitbun/langchain_test', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (4, 1536)  float32   None   
    ids      text     (4, 1)      str     None   
 metadata    json     (4, 1)      str     None   
   text      text     (4, 1)      str     None   

```

```python
# get embeddings numpy array
embeds = db.ds.embedding.numpy()

```

### 将本地数据集传输到云[#](#transfer-local-dataset-to-cloud "Permalink to this headline")


将已创建的数据集复制到云端。您也可以从云端传输到本地。

```python
import deeplake
username = "davitbun" # your username on app.activeloop.ai 
source = f"hub://{username}/langchain_test" # could be local, s3, gcs, etc.
destination = f"hub://{username}/langchain_test_copy" # could be local, s3, gcs, etc.

deeplake.deepcopy(src=source, dest=destination, overwrite=True)

```

```python
Copying dataset: 100%|██████████| 56/56 [00:38<00:00

```

```python
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/davitbun/langchain_test_copy
Your Deep Lake dataset has been successfully created!
The dataset is private so make sure you are logged in!

```

```python
Dataset(path='hub://davitbun/langchain_test_copy', tensors=['embedding', 'ids', 'metadata', 'text'])

```

```python
db = DeepLake(dataset_path=destination, embedding_function=embeddings)
db.add_documents(docs)

```



```python
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/davitbun/langchain_test_copy

```

```python
/

```

```python
hub://davitbun/langchain_test_copy loaded successfully.

```

```python
Deep Lake Dataset in hub://davitbun/langchain_test_copy already exists, loading from the storage

```

```python
Dataset(path='hub://davitbun/langchain_test_copy', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (4, 1536)  float32   None   
    ids      text     (4, 1)      str     None   
 metadata    json     (4, 1)      str     None   
   text      text     (4, 1)      str     None   

```

```python
Evaluating ingest: 100%|██████████| 1/1 [00:31<00:00
-

```

```python
Dataset(path='hub://davitbun/langchain_test_copy', tensors=['embedding', 'ids', 'metadata', 'text'])

  tensor     htype     shape     dtype  compression
  -------   -------   -------   -------  ------- 
 embedding  generic  (8, 1536)  float32   None   
    ids      text     (8, 1)      str     None   
 metadata    json     (8, 1)      str     None   
   text      text     (8, 1)      str     None   

```



```python
['ad42f3fe-e188-11ed-b66d-41c5f7b85421',
 'ad42f3ff-e188-11ed-b66d-41c5f7b85421',
 'ad42f400-e188-11ed-b66d-41c5f7b85421',
 'ad42f401-e188-11ed-b66d-41c5f7b85421']

```

